Boundary detection using vision-based feature mapping

ABSTRACT

A system configured to determine one or more boundaries of the space, for example, in 3D applications. In some cases, the system may generate point data including a set of safe points in a space and a set of unsafe points in the space, the space surrounding a user device of the system, generate a triangulation over a union of the set of safe points and the set of unsafe points, determine triangles of the triangulation that include at least one safe point and determine edges of determined triangles which are part of a single triangle that include at least one safe point. The system may then determine one or more boundaries of the space using the determined edges.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of and claims priority U.S. patentapplication Ser. No. 16/512,514, filed on Jul. 16, 2019 and entitled“Boundary Detection Using Vision-Based Feature Mapping,” which claimspriority to U.S. Provisional Application No. 62/698,632 filed on Jul.16, 2018 and entitled “Room Boundary Detection Using Vision-BasedFeature Mapping,” which are incorporated herein by reference in theirentirety.

BACKGROUND

The presence of three dimensional (3D) imaging and virtual realitysystems in today's world is becoming more and more common. In somecases, the imaging system or virtual reality system may be configured toallow a user to interact with the physical world or environmental in ameaningful manner. Many three-dimensional (3D) applications, especiallyvirtual reality (VR) outside-in tracked applications, may include aboundary associated with a space to keep users from running into realworld obstacles such as chairs, tables and walls.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 illustrates an example boundary detection system for generatingboundaries associated with a physical environment or space for a 3Dapplication according to some implementations.

FIG. 2 illustrates an example user engaged with a virtual environmentgenerated by a visualization system according to some implementations.

FIG. 3 is an example headset device according to some implementations.

FIG. 4 is a flow diagram showing an illustrative process for determininga determining and using boundaries in a 3D application according to someimplementations.

FIG. 5 is an example flow diagram showing an illustrative process fordetermining boundaries in an environment according to someimplementations.

FIGS. 6-12 illustrates example point data including safe points andunsafe points being utilized to determine boundaries in an environmentusing the techniques discussed with respect to FIGS. 1-5 according tosome implementations.

FIG. 13 is another example data flow diagram showing an illustrativedata flow for determining boundaries in an environment according to someimplementations

DETAILED DESCRIPTION

This disclosure includes techniques and implementations for utilizingcomputer vision to improve the detection for and generation of aboundary of a space. These boundaries may indicate or trace areas in thespace in which a user may safely move without colliding with walls,furniture, or other obstacles, including “internal” obstacles surroundedby accessible space.

Systems and methods according to this disclosure may use a 3Drepresentation of a space and tracked poses of a moving object to defineboundaries of obstacle-free “safe” areas which the moving object mayfreely access. For example, devices and systems according to thisdisclosure may direct a user to move around a space while lookingaround. While the user does so, the device may perform detection of realworld obstacles such as chairs, tables and walls and generate a boundarybased thereon. In some examples, the device may detect points of scenegeometry and determine points corresponding to known “safe” positionsrepresenting known points that are not inside obstacles. For example,safe points may be populated based on where the user or tracked objectis standing in the space.

The system may utilize the detected points of scene geometry todetermine unsafe points. More particularly, unsafe points may includedetections representing the geometry of objects in the space.

In some examples, the unsafe points may be filtered based on, forexample, each unsafe point's vertical position in the space, whether theunsafe points are contained inside a concave hull of the safe points,line of sight information from the device, or a combination thereof.Unsafe points may also be filtered by a confidence metric, or may bemodified by other geometric understanding of the space. For example,detected unsafe points may be overridden by detected wall planes ifnearby.

Some examples may utilize Delaunay triangulation to create a mesh ofsafe and unsafe regions in the space based on the safe points and unsafepoints. The resulting mesh may then be used to generate a boundaryaround the safe area. In other cases, a symbolic perturbation method maybe used to create the mesh of safe and unsafe regions.

For VR applications, systems and methods according to this disclosuremay provide a better user experience as the setup process disclosedherein may be performed by, for example, a headset device with apassthrough camera feed without a messy floor height setup process andwithout the user manually drawing the boundaries of the space.

Further, systems and methods in accordance with this disclosure mayallow for multiple boundaries (or boundary loops), which may allow theboundaries of the space to be setup with internal boundaries forobstacles such as tables, chairs, couches, and so on.

Systems and methods providing for such internal boundary loops may alsoutilize boundary visualization techniques which are different fromexternal boundary visualization. For example, the interaction withinternal obstacles such as tables, chairs, couches may differ fromexternal boundaries (e.g. walls) in that internal obstacles may beavoided and bypassed while external boundaries correspond to the outsideof the space and may not be bypassed. Thus, visualization techniques forinternal obstacles may balance obstacle avoidance and immersion. Forexample, visualization may be performed to maintain immersion whileallowing the user to avoid these obstacles even if they are below eyelevel. Some examples may use proximity and velocity to detect animpending collision with an internal boundary before the impendingcollision happens and make the obstacle corresponding to the internalboundary very obvious to the user in response to the impending collisiondetection. For instance, lines associated with the obstacle may bepresented on a display to the user over a currently viewed virtualenvironment.

In some examples, data representing the points in the scene geometry maybe generated by a visual-inertial simultaneous localization and mapping(SLAM) system that is, for example, included in a headset device worn bya user, but implementations are not limited by data source. Further,while implementations herein may relate to VR applications utilizing aninside-out visual simultaneous localization and mapping (SLAM) system,implementations are not limited to any application or input method.

Moreover, while implementations discussed herein relate to thecollection of data by a single device, implementations are not solimited. For example, the system may have a centralized process orreceiving information from user carried devices. In that case, multipleusers may carry such devices and safe point data and scene geometry datamay be captured by more than one device.

FIG. 1 illustrates an example boundary detection system 100 forgenerating boundaries associated with a physical environment or spacefor a 3D application according to some implementations. In theillustrated example, a user 102 may utilize a mobile device 104 tocapture data related to the physical environment or space 106 in whichthe user 102 is located.

The captured data (such as image data, depth data, red-green-blue data,orientation data, acceleration data, previously generated 3D models,etc.) may include unsafe point data related to barriers (e.g. walls) andobstacles, such as table 108, in the physical environment or space 106.In some examples, the unsafe point data captured by the mobile device104 may include feature points of objects, vertices in a meshreconstruction, point clouds from a depth sensor or LIDAR, or any otherdetection of points on objects in the space that may be consideredobstacles. In order to capture the unsafe point data, the mobile device104 may include one or more sensor components to capture informationabout the physical environment in which the user 102 and mobile device104 are located from substantially the perspective or view of the user102 and/or the mobile device 104. Thus, in some cases, the mobile device104 may capture the information about the physical environment based ona field of view 110 substantially similar to a field of view of themobile device 104 and/or the user 102 as the user 102 moves about thephysical environment.

In addition, the mobile device 104 may determine safe point data for thespace. In some implementations, the mobile device 104 may recordlocation data representing where the user moved in the space 106 duringthe capture of the data. For example, safe point data may be generatedfrom a headset pose of the user 102 and mobile device 104 during theboundary setup process. The safe point data may be sampled from themobile device 104 position at set time and distance intervals. In aparticular example, safe point data may be sampled every 0.5 seconds or10 centimeters travelled, whichever samples more frequently giveninstantaneous velocity.

The mobile device 104 may then process and filter the captured data. Theprocessed and filtered data may then be utilized to generate boundariesfor the 3D application. For example, the mobile device 104 may perform aDelaunay triangulation on a union of detected unsafe points (e.g. in theunsafe point data) and known-safe positions (e.g. from the safe pointdata), then process the resulting mesh into a safe boundary of space106. The safe boundaries may trace areas of the space 106 where the user102 can safely move without colliding with walls, furniture, or otherobstacles, including “internal” obstacles (e.g. around table 108)surrounded by accessible space.

Once the boundaries have been determined, the mobile device 104 mayenter an operational mode in which, for example, a virtual environmentis displayed to the user 102. As the user 102 interacts with the virtualenvironment, the boundaries may be utilized to provide indications orwarnings to the user 102 of potential collisions with barriers orobstacles in the space 108. In this way, the user may avoid collisionsand potential injuries without engaging in a time-consuming manual setupprocess.

FIG. 2 illustrates an example user 200 engaged with a virtualenvironment generated by a visualization system 202 according to someimplementations. In some implementations, the visualization system 202may include an electronic device 204 and a headset device 206. Forinstance, the visualization system 202 may be modular, such that theheadset device 206 may be compatible with a verity of electronic devices204 having a virtual environment application installed. In someinstances, the visualization system 202 may be configured to provide anenclosed environment from which the virtual environment may be viewed,while in other instances, the visualization system 202 may be an openair design. In one particular implementation, the visualization system202 may be convertible from an open air system to an enclosed system. Inother examples, the visualization system 202 may be a self-containedunit that includes processing resources, sensors or image components,displays, lenses, projects, etc. In this example, the visualizationsystem 202 may take other forms, such as a handheld system or otherdesigns that may or may not fit on the user's 200 head.

As discussed above, in one example, the electronic device 204 may be aportable electric device, such as a tablet, netbook, laptop, cell phone,mobile phone, smart phone, etc. that includes processing and storageresources, such as processors, memory devices, and/or storage devices.The headset 206 may include components configured to secure theelectronic device 204 in a manner viewable by the user 200 as well assensors, measurement units or devices, and image components forcapturing images and/or video from a physical environment. In somecases, the headset 206 may also include projectors to, for instance,project a grid or pattern within the physical environment to assist withdetermining positions and surfaces of the physical environment. Forinstance, the visualization system 202 may be configured to performoperations, such as structured light, to detect objects in the space106, such as stationary objects (e.g., walls, furniture, etc.). In somespecific examples, the headset 206 may also be equipped with specializedviewing lenses that together with the display of the electronic device202 cause the user 200 to experience the virtual environment in 3D.

FIG. 3 is an example headset device 300 according to someimplementations. As described above, the headset device 300 may be acombination of an electronic device and a headset or be a self-containedunit. In the current example, the headset device 300 may include sensorcomponents 302 for capturing data, such as feature points of objects,vertices in a mesh reconstruction, point clouds from, for instance, adepth sensor or LIDAR, or any other detection of points on objects inthe space that may be considered obstacles, from a physical environment.For example, the sensor components 302 may be positioned to capturedepth or scene geometry data from substantially the same perspective asthe user (e.g., a position proximate the user's eyes or head) in orderto incorporate the data associated with the captured data into thevirtual environment. The sensor components 302 may be of various sizesand quality. For instance, the sensor components 302 may include one ormore depth and geometry sensors such as a stereo camera system, amonocular SLAM system, a depth camera, and a LIDAR or other rangingsystem. In general, the sensor components 302 may each include variouscomponents and/or attributes.

In some cases, the headset device 300 may include one or moremeasurement units 304 that may determine orientation data of the headsetdevice 300 (e.g., acceleration, angular momentum, pitch, roll, yaw, etc.of the headset device 300). The measurement units 304 may include one ormore IMUs, one or more accelerometers, one or more gyroscopes, one ormore magnetometers, and/or one or more pressure sensors, as well asother sensors. In one particular example, the measurement units 304 mayinclude three accelerometers placed orthogonal to each other, three rategyroscopes placed orthogonal to each other, three magnetometers placedorthogonal to each other, and a barometric pressure sensor.

The headset device 300 may also include one or more communicationinterfaces 306 configured to facilitate communication between one ormore networks, one or more cloud-based management system, and/or one ormore physical objects. The communication interfaces 306 may alsofacilitate communication between one or more wireless access points, amaster device, and/or one or more other computing devices as part of anad-hoc or home network system. The communication interfaces 306 maysupport both wired and wireless connection to various networks, such ascellular networks, radio, WiFi networks, short-range or near-fieldnetworks (e.g., Bluetooth®), infrared signals, local area networks, widearea networks, the Internet, and so forth.

The visualization system 300 may also include one or more processors308, such as at least one or more access components, control logiccircuits, central processing units, or processors, as well as one ormore computer-readable media 310 to perform the function associated withthe virtual environment. Additionally, each of the processors 308 mayitself comprise one or more processors or processing cores.

Depending on the configuration, the computer-readable media 310 may bean example of tangible non-transitory computer storage media and mayinclude volatile and nonvolatile memory and/or removable andnon-removable media implemented in any type of technology for storage ofinformation such as computer-readable instructions or modules, datastructures, program modules or other data. Such computer-readable mediamay include, but is not limited to, RAM, ROM, EEPROM, flash memory orother computer-readable media technology, CD-ROM, digital versatiledisks (DVD) or other optical storage, magnetic cassettes, magnetic tape,solid state storage, magnetic disk storage, RAID storage systems,storage arrays, network attached storage, storage area networks, cloudstorage, or any other medium that can be used to store information andwhich can be accessed by the processors 308.

Several modules such as instructions, data stores, and so forth may bestored within the computer-readable media 310 and configured to executeon the processors 308. For example, as illustrated, thecomputer-readable media 310 store device pose estimation instructions312, feature detection instructions 314, boundary detection instructions316 and boundary visualization instructions 318.

The device pose estimation instructions 312, when executed by theprocessor 308 may cause the processor 308 to perform operations todetermine the pose of the headset device 300 (e.g. the position andorientation of the headset device 300 in relation to the space orvirtual environment). For instance, the device pose estimationinstructions 312 may cause the processor 308 to determine the pose ofthe headset device 300 every 0.5 seconds or 10 centimeters travelled,whichever samples more frequently given instantaneous velocity. In someexamples, the pose of the headset may be determined by the device poseestimation instructions 312 based on information provided by themeasurement units 304. Once determined, the location and orientation ofthe headset device 300 represented by the pose may be stored. Someexamples may utilize the stored location and orientation of the headsetdevice 300 to determine safe point data which may be used in determiningboundaries for the space.

The feature detection instructions 314 may be configured to determinepoint data using information from sensor components 302 and datagenerated by the pose estimation instructions 312. As discussed above,the point data may include unsafe point data related to barriers andobstacles, such as table 108, in the physical environment or space 106.In some examples, the unsafe point data determined by the featuredetection instructions 314 may include feature points of objects,vertices in a mesh reconstruction, point clouds from a depth sensor orLIDAR, or any other detection of points on objects in the space that maybe considered obstacles.

In addition, the feature detection instructions 314 may determine safepoint data in the space. In some implementations, the feature detectioninstructions 314 may utilize the stored pose data representing theorientation of the headset and where the user moved in the space duringthe setup process to determine safe points that represent areas in thespace that are accessible to the user. As mentioned above, the pose datamay be sampled and safe point data may be determined by the featuredetection instructions 314 at set time and distance intervals. In aparticular example, safe point data may be determined every 0.5 secondsor 10 centimeters travelled, whichever provides more frequentinformation given instantaneous velocity.

The feature detection instructions 314 may then process and filter thepoint data (e.g. the unsafe point data and safe point data). Forexample, the unsafe point data may be filtered based on proximity todetected vertical planes, metadata associated with detected unsafepoints (e.g. surface area, extent, and orientation of planes on whichthe unsafe points lie, length and orientation of line features, orconfidence metrics), vertical positioning of the unsafe points, and bygenerating a concave hull using safe points and removing surroundedunsafe points inside the concave hull.

In addition, the feature detection instructions 314 may generateadditional safe points based on line of sight information determinedfrom the line of sight between a safe point where the headset device 300is located and the location of unsafe points captured from the safepoint (e.g. the area of the space corresponding to the unobstructed lineof sight may be determined safe).

The feature detection instructions 314 may then project the remainingunsafe points representing obstacles and all of the safe points to ahorizontal plane.

The boundary detection instructions 316 may utilize the processed andfiltered point data to generate boundaries for the space. For example,the boundary detection instructions 316 may perform a Delaunaytriangulation on a union of detected unsafe points (e.g. in the unsafepoint data) and known-safe positions (e.g. from the safe point data),then process the resulting mesh into safe boundaries. The safeboundaries may trace areas of the space where the user can safely movewithout colliding with walls, furniture, or other obstacles. Theboundaries may include internal boundaries around obstacles surroundedby accessible space (e.g. furniture or within the space).

The boundary visualization instructions 318 may utilize the boundariesdetermined by the boundary detection instructions 316 during anoperational mode in which, for example, a virtual environment isdisplayed to the user of the headset device 300. As the user interactswith the virtual environment, the boundaries may be utilized to provideindications or warnings to the user 102 of potential collisions withbarriers or obstacles in the space 108.

FIGS. 4-5 and 13 are flow diagrams illustrating example processesassociated with determining boundaries in an environment according tosome implementations. The processes are illustrated as a collection ofblocks in a logical flow diagram, which represent a sequence ofoperations, some or all of which can be implemented in hardware,software or a combination thereof. In the context of software, theblocks represent computer-executable instructions stored on one or morecomputer-readable media that, which when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, encryption, deciphering, compressing, recording, datastructures and the like that perform particular functions or implementparticular abstract data types.

The order in which the operations are described should not be construedas a limitation. Any number of the described blocks can be combined inany order and/or in parallel to implement the process, or alternativeprocesses, and not all of the blocks need be executed. For discussionpurposes, the processes herein are described with reference to theframeworks, architectures and environments described in the examplesherein, although the processes may be implemented in a wide variety ofother frameworks, architectures or environments.

FIG. 4 is an example flow diagram showing an illustrative process 400for determining a determining and using boundaries in a 3D applicationaccording to some implementations. As discussed above, described hereinis a visualization system that may be configured to determine boundariesin an environment or space. For example, a headset device worn by usermay capture data related to the scene geometry of a space as the userwalks through safe or accessible areas of the space. The captured dataand safe area data may be utilized to determine boundaries for the 3Dapplication.

At 402, the visualization system may display a request to the user tomove around the space while varying the user's view angle. At 404, thevisualization system may detect features of the space as safe and unsafepoints as the user moves around the space while varying the users viewangle. For example, the visualization system may include or be equippedwith various sensor components and measurement components which mayprovide data for use in determining the safe and unsafe points. Thevisualization system may also perform filtering and other processes onthe detected safe and unsafe points. The visualization system may thenproject the remaining unsafe points representing obstacles and all ofthe safe points to a horizontal plane. In some cases, points may beremoved if the points are outside of a threshold of the horizontalplane, result in physically impossible or highly unlikely in a physicalenvironment, etc.

At 406, the visualization system may perform boundary detection on thedetected features. For example, the visualization system may perform aDelaunay triangulation on a union of detected unsafe points (e.g. in theunsafe point data) and known-safe positions (e.g. from the safe pointdata), then process the resulting mesh into safe boundaries. Forexample, the mesh or point cloud may be merged into regions, that may bein turn merged into larger regions forming the safe and unsafeboundaries. The safe boundaries may trace areas of the space where theuser can safely move without colliding with walls, furniture, or otherobstacles. The boundaries may include internal boundaries aroundobstacles surrounded by accessible space (e.g. furniture or within thespace).

At 408, the visualization system may utilize the boundaries determinedat 406 in, for example, in a virtual reality simulation. For example, asthe user interacts with the virtual reality simulation, the boundariesmay be utilized to provide indications or warnings to the user ofpotential collisions with barriers or obstacles in the space.

FIG. 5 is an example flow diagram showing an illustrative process 500for determining boundaries in an environment according to someimplementations. More particularly, process 500 provides additionaldetails related to the operation of some examples of the systems andmethods disclosed herein. During the discussion of process 500, FIGS.6-12 may be discussed to provide a visual example of the operations.

At 502, the visualization system may capture sensor data from the spaceas the user moves around. As discussed above, the user may be instructedby the visualization system to perform such actions.

At 504, the visualization system may generate point data describing bothsafe and unsafe points in a scene. As discussed above, “unsafe” pointsmay be based on detections representing geometry of objects in the scenewhich may correspond to obstacles or barriers. “Safe” points may belocations or points known to be free of obstacles in the context of thescene; these may be based on the tracked positions of a headset.

As mentioned above, the unsafe points may be sparse feature points onobjects, vertices in a mesh reconstruction, point clouds from a depthsensor or LIDAR, or any other data that may represent points of objectsin space that could be considered obstacles. In an exampleimplementation of a sparse SLAM system which may find 3D feature points,lines, and planes on world geometry, the visualization system maygenerate unsafe points using detected lines and detected planes in worldspace.

The visualization system may utilize detected lines to generate unsafepoints at set distances along the line segment, such as every 10 cmalong the line, and at both endpoints. In some examples, these pointsmay be removed from the set of unsafe points if the points within athreshold (e.g. 10 cm) of a detected plane, as the confidence andgeometric accuracy of the plane may be higher than that of the detectedline.

The visualization system may utilize detected planes in a similarmanner. In particular, the visualization system may generate unsafepoints at set distances every 0.1 meters along intersections of thedetected plane and a horizontal plane(s) at height(s) deemed reasonablefor obstacles (e.g. a height that will not be filtered out as anon-obstacle by the unsafe point filtering process below).

The visualization system may generate safe points based on the user'sheadset pose during the setup process. As mentioned above, the headsetpose may include the location of the headset within the space and theorientation of the headset. The safe points may be sampled from theheadset device position at set time and distance intervals (e.g. every0.5 seconds or 10 centimeters travelled, whichever samples morefrequently given instantaneous velocity).

FIG. 6 provides an illustration of a set of points 600 that may begenerated by visualization systems according to some implementations.For the purposes of the discussion of FIGS. 6-12, the set of points mayrepresent point data captured from a portion of the space 106 of FIG. 1.In particular, the set of points may correspond to point data capturedin the area of the room shown in FIG. 1 below the wall mounted displayand including the table 108. The illustrated view of FIGS. 6-12 mayrepresent a top-down view perpendicular to the floor of the space 106.While shown in two-dimensions for ease of illustration understanding,the point data of the set of points may also include vertical positiondata.

As illustrated, empty circles, such as point 602, may represent safepoints in the space 106 and filled circles, such as 604, may representunsafe points which may correspond to detected scene geometry of thespace 106.

Returning to FIG. 5, at 506, the visualization system may filter andprocess the set of unsafe and safe points described by the point data.In some cases, one or more of the points detected by a vision system maynot be useful in the context of boundary generation. For example, anunsafe point may be on the ceiling or floor, rather than on furniture orother obstacles or barriers. The visualization system may utilize anumber of filtering methods to reduce the set of points consideredunsafe. In addition, the safe points may be inferred or estimated basedon known information such as line of sight. The visualization system mayadd these inferred or estimated safe points to the set of points. Thediscussion of filtering and processing the set of points provided belowis discussed in the context of FIG. 7.

FIG. 7 provides an illustration of the set of points 700 that may begenerated by visualization systems according to some implementations.More particularly, the set of points 700 may represent the set of points600 following filtering and processing at 506. Unsafe points selectedfor removal by the visualization system are shown with crosshatch (e.g.points 702-710).

In a first filtering operation, the visualization system may utilizevertical position of unsafe points with respect to a ground plane tofilter the unsafe points. For example, the visualization system maydetermine that unsafe points located below the ground plane, or lessthan a threshold above the ground plane may be removed. In theillustrated set of points 700, point 702 may represent an unsafe pointdetermined to be below the ground plane or less than a threshold abovethe ground plane.

In addition, the visualization system may also filter out points thatmay be more than a human-reach threshold above the ground plane, or morethan a threshold distance above the positions of the, for example,headset device of the visualization system during the setup process. Inthe illustrated set of points 700, point 704 may represent an unsafepoint detected from the light fixture substantially above the table 108.In some examples, the visualization system may determine that the point704 is more than a human-reach threshold above the ground plane, or morethan a threshold distance above the positions of the headset device.

In a second filtering operation, the visualization system may utilizemetadata associated with detected unsafe points to remove unsafe points.For example, metadata such as the surface area, extent, and orientationof any planes on which the unsafe points lie, the length and orientationof line features, or confidence metrics may be used to decide whether aparticular unsafe point should remain in the set of points 700. In theillustrated set of points 700, points 706A and 706B may represent pointsremoved based on metadata.

In a third filtering operation, the visualization system may draw aconcave hull 708 around the detected safe points. In some examples, aconvex hull (also known as a convex envelope or a convex closure) for aset X of points in the Euclidean plane or in a Euclidean space (or, moregenerally, in an affine space over the reals) may be the smallest convexset that contains X. One of ordinary skill in the art would understandhow to draw a concave hull around the detected safe points in view ofthis disclosure.

The visualization system may then determine whether any of the unsafepoints inside the concave hull 708 are isolated or surrounded by safepoints. In the illustrated set of points 700, unsafe point 710 may bedetermined to be surrounded by safe points. The visualization system mayremove the unsafe points determined to be surrounded. As such, point 710may be marked for removal. In some examples, the removal of unsafepoints surrounded by safe points within the concave hull may allow thevisualization system to remove spurious detections without removinginternal boundaries that represent obstacles such as the table 108 inspace 106.

The above described filtering operations are merely examples and variousimplementations may include additional or alternative filteringprocesses.

As discussed above, the visualization system may generate additionalsafe points which may be inferred or estimated based on conditionalinformation available to the visualization system.

For example, the visualization system may generate additional safepoints using line of sight information. More particular, if the headsetdevice is oriented towards an unsafe point (e.g. representing anobstacle or barrier) and there is open space between the headset deviceand the unsafe point, the visualization system may insert additionalsafe points along the line of sight between the headset device in theunsafe point. The inclusion of the additional safe points may improvethe resulting boundaries for the space. For example, the additional safepoints may allow boundaries generated based on the additional safepoints to extend closer to the real geometry of the space and may reduceamount of the user must traverse the space while changing theorientation of the headset device. In the illustrated set of points 700,items 712-716C relate to this process.

More particular, the visualization system may determine that the headsetdevice was positioned at safe point 712 when the visualization systemdetected an unsafe point in the direction indicated by arrow 714 whichrepresents the clear line of sight to the unsafe point. Based on theclear line of sight 714, the visualization system may add safe points716A-716C.

The above described operation to generate additional safe points ismerely an example and various implementations may include additional oralternative processes for generating additional safe points based oninferences from known data.

Returning to FIG. 5, at 508, the visualization system may project theprocess and filtered set of points to a horizontal plane. FIG. 8provides an illustration of the set of points 800 that may be generatedby visualization systems according to some implementations. Moreparticularly, the set of points 800 may represent the set of points 800following removal of filtered unsafe points (e.g. points 702-710) andthe inclusion of additional safe points (e.g. points 716). In addition,the set of points 800 may have been projected to a horizontal planeparallel to the floor of space 106.

Next, at 510, the visualization system may generate a Delaunaytriangulation over the union of the sets of projected safe points andunsafe points. The resulting Delaunay mesh may have the advantage ofconnecting nearby points in a way that preserves boundaries, but whichmay also allow easy isolation of boundary points by surrounding themwith safe points.

A Delaunay triangulation for a given set P of discrete points in a planemay be a triangulation DT(P) such that no point in P is inside thecircumcircle of any triangle in DT(P). Delaunay triangulations maymaximize the minimum angle of all the angles of the triangles in thetriangulation and may avoid sliver triangles. One of ordinary skill inthe art would understand how to implement a Delaunay triangulation forthe safe and unsafe points in view of this disclosure.

FIG. 9 provides an illustration of the set of points 900 that may begenerated by visualization systems according to some implementations.More particularly, the set of points 900 may represent the set of points800 following the Delaunay triangulation at 510. As illustrated,connectors 902 connect the safe and unsafe points into a mesh inaccordance with the triangulation algorithm. It should be noted that theinterconnections illustrated in FIGS. 9-12 are approximated andillustrative and not intended to represent an exact triangulation thatwould be generated for the set of points 900.

Returning to FIG. 5, at 512, the visualization system may filter theDelaunay mesh to include triangles that may be “safe” triangles. As usedherein, “safe triangles” may refer to triangles that contain at leastone safe point. As such, the Delaunay mesh may be filtered by selectingall triangles that contain at least one safe point and ignoringtriangles that do not contain a safe point. The resulting “safe mesh”may include boundaries that describe the boundaries of the detected safespace.

FIG. 10 provides an illustration of the set of points 1000 that may begenerated by visualization systems according to some implementations.More particularly, the set of points 1000 may represent the set ofpoints 900 following the filtering of the Delaunay mesh at 512 to selectthe safe triangles (e.g. to generate a safe mesh). As shown in FIG. 10,triangles that include a safe point, such as triangle 1002, remainselected (e.g. shown as solid lines) while triangles that do not includea safe point, such as triangle 1004, are ignored (e.g. shown as dashedlines).

Returning to FIG. 5, at 512, the visualization system may locate theboundaries of the safe area by finding non-manifold edges in the safemesh (e.g. edges that are only used in one selected triangle (e.g. inone safe triangle)). In some examples, the visualization system may markall edges that are only used in one selected triangle (e.g. in one safetriangle). The unmarked edges of the safe mesh may then be ignored orremoved and the marked edges may be sorted to line up to find theboundaries of the space. This resulting safe mesh may include allboundaries of the safe area of the space, including internal boundariesaround objects.

FIG. 11 provides an illustration of the set of points 1100 that may begenerated by visualization systems according to some implementations.More particularly, the set of points 1100 may represent the set ofpoints 1000 following the locating and marking of the non-manifold edgesof the safe mesh (shown as bold lines), such as edge 1102.

FIG. 12 provides an illustration of the set of points 1200 that may begenerated by visualization systems according to some implementations.More particularly, the set of points 1200 may represent the set ofpoints 1100 following the ignoring or removal of the unmarked edges ofthe safe mesh (shown as dashed lines), such as edge 1202. Further, theboundaries of the space 106 may now be seen from the connected markededges. For example, referring to FIG. 1, the boundary 1204 may representthe portion of the wall in FIG. 1 including the wall mounted display.Similarly, the internal boundary 1206 may represent the table 108.

While the boundaries 1204-1206 are illustrated as being straight andrelatively precise, depending on the density of the unsafe point data,visualization systems according to this disclosure may generateboundaries that deviate from the real-world geometry of the space 106.For example, the clear line of sight 714 allowed for points 716A-716 Cto be generated and to represent the safe space between the table 108and the wall of the space 106. In cases in which the use does not lookin the direction indicated by arrow 714 from point 712, the boundary1206 may bulge out and include the area of boundary 1206 and the safearea between the objects. In another example, due the lack of a clearline of sight from the safe point 712 to unsafe points on the far sideof the table 108 (e.g. the table blocks the line of sight as usedherein), the boundary for the table 108 may extend away from the actualgeometry of the table on the far side to, for example, combine with aboundary of a nearby chair (not shown). To avoid such deviation, theuser may walk around the far side of the table 108 or provides a clearline of sight for the inference of safe points on the far side of thetable 108.

Once the visualization system has determined the boundaries 1204-1206,the visualization system may enter an operational mode. Duringoperation, the boundaries may be utilized by visualization instructionswhich may update or warn the user regarding the user's relative locationwith respect to the detected boundaries (e.g. as the user moves aroundthe space while interacting with the virtual environment).

In some virtual reality implementations, the visualization system mayutilize external and internal boundaries to create a bettervisualization technique for boundaries than the existing techniques. Thevisualization system may begin by visualizing internal boundaries asgrid meshes with height attached. For example, the table 108 may beabout 20 inches high and, in turn, boundary 1206 may be visualized as agrid mesh box with a virtual height of about 20 inches.

In some examples, the visualization system may generate the virtualenvironment to such that the mesh fades away as the user leaves itsproximity (e.g. moves more than one meter from the boundary), and fadesin as the user approaches the boundary. In some cases, when the usergets very close (e.g. 0.5 meters), the visualization system may measurethe velocity and field-of-view and project the box mesh upwards so it isalways visible and it becomes opaque if the user is moving quickly. Inthis way, the visualization system may create an extended mesh that isnon-intrusive during normal operation, but becomes noticeable when theuser is nearing a collision.

In some implementations, the visualization system may perform variousupdates to the safe area during operation. For example, thevisualization system may continue to collect point data regarding bothunsafe points and safe points while the user is operating within the 3Dapplication and use the additional point data to refine the boundaries.These updates may be activated manually, be performed periodically or beperformed in real time as additional data is collected. Moreover, thevisualization system may preserve the safe area and boundary informationbetween sessions and perform updates at start up. A full determinationof the safe boundaries may be triggered manually or performed upondegeneration that the visualization system is no longer located in thesame space.

In another example, the visualization system may provide for user inputregarding detected objects in the space. For example, the user mayindicate that various objects are either permanent or temporary. Basedon such metadata, the visualization system may selectively ignore orotherwise mark the nature of the obstacle or barrier in the space.

FIG. 13 is an example data flow diagram showing an illustrative dataflow 1300 for determining boundaries in an environment according to someimplementations. More particularly, data flow 1300 provides additionaldetails related to the operation of some examples of the systems andmethods disclosed herein

At 1302, the data flow may to determine the pose of the headset device.The pose data may include, for example, the position and orientation ofthe headset device in relation to the space or virtual environment.

At 1304, the data flow may collect sparse data including data related todetected lines 1306, detected vertical planes 1308 and detected groundsplanes 1310.

Next, at 1312, the data flow may determine safe point data in the space.As discussed above, the safe point data may include locations traversedby the user while wearing the headset device during the setup process.

At 1314, the data flow may detect obstacles in the space. Using thetechniques discussed above with respect to FIGS. 1-12, the data flow mayutilize the collected sparse data to generate unsafe point data relatedto obstacles and barriers in the space.

At 1316, the data flow may include filtering and processing the pointdata. Using the techniques discussed above regarding FIGS. 1-12, thefiltering and processing functions may include functions for filteringunsafe point data based on proximity to detected vertical planes 1318,filtering unsafe point data based on metadata 1320, filtering unsafepoint data based on vertical positioning of the unsafe points 1322,generating additional safe point data based on line of sight information1324 and filtering unsafe point data based on safe concave hull analysis1326.

At 1328, the data flow may project the processed and filtered point datafrom a 3D space onto a 2D plane. The projected point data may then beprocessed using a Delaunay triangulation to generate a Delaunay mesh at1330. Finally, at 1332, the data flow may generate boundaries based onthe Delaunay triangulation using the techniques discussed above withrespect to FIGS. 1-12.

The aforementioned systems and techniques include a set of illustrativetechniques for determining and using boundaries in a 3D application suchas in virtual reality simulations. However, other techniques may beemployed to accomplish similar results.

For example, while the boundary detection technique shown and describedabove utilized projection from 3D to 2D prior to performing Delaunaytriangulation, implementations are not so limited.

For example, implementations may determine the boundary in threedimensions. In some examples, the technique may determine athree-dimensional boundary is similar in principle, with differences toaccount for additional dimensionality. First, the points may not beprojected to a 2D plane, instead remaining 3D point locations. Second,3D Delaunay triangulation may be performed instead of a 2Dtriangulation, generating tetrahedral simplices. Third, tetrahedralsimplices rather than 2D triangles may be marked as “safe” if thetetrahedral simplices contain at least one “safe” point. Fourth, themesh may be filtered to include only tetrahedrons that are marked assafe. Next, the boundaries of safe regions may be 3D surfaces consistingof triangles, rather than 2D line edges and may represent the boundariesof the volumes consisting of these tetrahedrons that are marked as safe.Finally, the safe boundaries may be a 3D surface consisting of trianglefaces that may each be part of only one tetrahedron, rather than edgesthat may be a part of only one triangle.

Further, the same approach may be mathematically generalized to N>=2dimensions. First, safe and unsafe points may be determined in Ndimensions. Second, a N-dimensional Delaunay triangulation may beperformed on these points, producing simplices of N+1 points in Ndimensions. Next, simplices of N+1 points may be marked as safe if thesimplices contain at least one safe point. Finally, boundaries of thesafe region may consist of a “surface mesh” consisting of elements of Npoints, which may represent the set of mesh elements bordered by onlyone simplex.

Implementations considering higher dimensionality may have utility wheretime or other parameters are variables in a given application. Forexample, if a scene or detections within a scene are known to changewith respect to time, the use of triangulation on 4D points (3D+time)may be advantageous. The visualization system may then evaluate sectionsor samples of the resulting 4D boundaries at specific times to visualizehow the 3D safe boundary may change with respect to time.

In some implementations, the visualization systems described herein mayutilize deep learning aided plane detection. In some cases, to enableimproved automatic setup for spaces or other environment-dependentapplications in AR/VR, the visualization system may recover the geometryof the space. While using sparse features or sparse lines as discussedabove is an appropriate methods, planar regions more directly correspondto the major structural elements in a typical interior space. Deeplearning aided plane detection may provide a method to robustly detectthese planar regions which may aid in the setup process.

In this process, a deep neural net may be trained to recover surfacenormals from visible image inputs and, optionally, a map ofdiscontinuities in depth. Next, a sequence of images (stereo ormonocular) and IMU data may then processed. More particularly, ascale-accurate trajectory may be recovered by the visual-inertial (orstereo) SLAM. Next, the trajectory may be used in multi-view depthestimation using a framework like that in Large-Scale Direct SLAM(LSD-SLAM). After some number of frames and when scale-accuratesemi-dense depth has been calculated, images from the sequence may bepassed into a deep neural network trained to infer surface normal and,optionally, depth discontinuities. A flood-fill style algorithm may thenbe used to find near-constant segments in the surface normals,optionally using the inferred depth discontinuities to avoid fillingacross discontinuities.

Each plane segment is then processed. In particular, the segment normalmay be estimated using an outlier-robust average, or a simple average.Next, the semi-dense depth map values may be optionally transformed toalign with the plane normals, such that depth values which may becoplanar to the segment's normal may become constant across the image.

Then, the plane may be processed further using either traditionalmethods or deep learning methods.

If traditional methods are used, the planar region may be used as a seedto search for depth values consistent with the plane. Values inside theregion may be preferred, and if they are self-consistent, a robust meanmay be returned as the plane's inferred depth. Next, if depth valueswithin the planar region do not meet a coverage threshold, then a floodfill/dilation may be used to expand the planar region, potentially usingthe inferred depth discontinuities to avoid expanding beyond the trueunderlying planar region. As the region is expanded in severaliterations, each time the visualization system may check again if thedepth inside the expanded plane region meet a self-consistent thresholdsufficiently for the plane depth to be estimated.

On the other hand, if deep learning is used, the chosen assumed-planarregion (and optionally surrounding planar regions), semi-dense depthmap, and optionally the inferred discontinuity map, may be given by thevisualization system as input to a deep neural net as input. The deepneural net receiving the input may be trained to recover the true depthof the planar region and a more precise planar region (expanded to trulyfit the real underlying plane, even if the flood-fill output waspartial).

Finally, the returned planar region that includes the recovered depth isused as part of a subsequent system. In some systems, the planar regionsmay be merged together over time using geometric consistency checks.

Although the subject matter has been described in language specific tostructural features, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features described. Rather, the specific features are disclosedas illustrative forms of implementing the claims.

What is claimed is:
 1. A system comprising: one or more processors;non-transitory computer-readable media storing computer-executableinstructions, which when executed by the one or more processors causethe one or more processors to: generate point data including a set ofsafe points in a space and a set of unsafe points in the space, thespace surrounding a user device of the system; generate a triangulationover a union of the set of safe points and the set of unsafe points;determine triangles of the triangulation that include at least one safepoint; determine edges of determined triangles which are part of asingle triangle that includes at least one safe point; and determine oneor more boundaries of the space using the determined edges.